We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

Scaling up language models has led to unprecedented performance gains, but little is understood about how the training dynamics change as models get larger. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors? In this paper, we analyze the intermediate training checkpoints of differently sized OPT models (Zhang et al.,2022)--from 125M to 175B parameters--on next-token prediction, sequence-level generation, and downstream tasks. We find that 1) at a given perplexity and independent of model sizes, a similar subset of training tokens see the most significant reduction in loss, with the rest stagnating or showing double-descent behavior; 2) early in training, all models learn to reduce the perplexity of grammatical sequences that contain hallucinations, with small models halting at this suboptimal distribution and larger ones eventually learning to assign these sequences lower probabilities; 3) perplexity is a strong predictor of in-context learning performance on 74 multiple-choice tasks from BIG-Bench, and this holds independent of the model size. Together, these results show that perplexity is more predictive of model behaviors than model size or training computation.

语言模型既展示了定量的改进，又展示了新的定性功能，随着规模的增加。尽管它们具有潜在的变革性影响，但这些新能力的特征却很差。为了为未来的研究提供信息，为破坏性的新模型能力做准备，并改善社会有害的效果，至关重要的是，我们必须了解目前和近乎未来的能力和语言模型的局限性。为了应对这一挑战，我们介绍了超越模仿游戏基准（Big Bench）。 Big Bench目前由204个任务组成，由132家机构的442位作者贡献。任务主题是多样的，从语言学，儿童发展，数学，常识性推理，生物学，物理学，社会偏见，软件开发等等。 Big-Bench专注于被认为超出当前语言模型的功能的任务。我们评估了OpenAI的GPT型号，Google内部密集变压器体系结构和大型基础上的开关稀疏变压器的行为，跨越了数百万到数十亿个参数。此外，一个人类专家评估者团队执行了所有任务，以提供强大的基准。研究结果包括：模型性能和校准都随规模改善，但绝对的术语（以及与评估者的性能相比）；在模型类中的性能非常相似，尽管带有稀疏性。逐渐和预测的任务通常涉及大量知识或记忆成分，而在临界规模上表现出“突破性”行为的任务通常涉及多个步骤或组成部分或脆性指标；社交偏见通常会随着含糊不清的环境而随着规模而增加，但这可以通过提示来改善。

预先训练的蒙版语言模型通过将下游任务作为文本填充来成功执行几次学习。但是，作为全镜头环境中的强大替代方案，诸如Electra之类的判别预训练模型不适合范式。在这项工作中，我们调整了基于及时的几次学习来进行电信，并表明它在广泛的任务中优于蒙面的语言模型。Electra是预先训练的，以区分令牌是产生还是原始。我们自然地将其扩展到基于迅速的几次学习，通过培训来评分目标选项的原创性，而无需引入新参数。我们的方法很容易适应涉及多token预测的任务，而无需额外的计算开销。分析表明，Electra学习分布与下游任务更好。

Recent developments of dense retrieval rely on quality representations of queries and contexts coming from pre-trained query and context encoders. In this paper, we introduce TouR (test-time optimization of query representations), which further optimizes instance-level query representations guided by signals from test-time retrieval results. We leverage a cross-encoder re-ranker to provide fine-grained pseudo labels over retrieval results and iteratively optimize query representations with the gradient descent method. Our theoretical analysis reveals that TouR can be viewed as a generalization of the classical Rocchio's algorithm for pseudo relevance feedback, and we present two variants leveraging psuedo labels as either hard binary or soft continuous labels. We first apply TouR on phrase retrieval with our proposed phrase re-ranker. On passage retrieval, we demonstrate its effectiveness with an off-the-shelf re-ranker. TouR improves the end-to-end open-domain QA accuracy significantly, as well as passage retrieval performance. Compared to re-ranker, TouR requires a smaller number of candidates, and achieves consistently better performance and runs up to 4x faster with our efficient implementation.

Recent work has improved language models (LMs) remarkably by equipping them with a non-parametric memory component. However, most existing approaches only introduce mem-ories at testing time or represent them using a separately trained encoder, resulting in suboptimal training of the language model. In this work, we present TRIME, a novel yet simple training approach designed for training LMs with memory augmentation. Our approach uses a training objective that directly takes in-batch examples as accessible memory. We also present new methods for memory construction and data batching, which are used for adapting to different sets of memories--local, long-term, and external memory--at testing time. We evaluate TRIME on multiple language modeling and machine translation benchmarks and show that it is able to achieve significant improvements across all the settings. Concretely, TRIME reduces the perplexity from 18.70 to 15.37 on WIKITEXT-103, by effectively leveraging a large memory set from the training corpus. Compared to standard LM training, TRIME adds negligible computational overhead and is compatible with different neural architectures, making it a versatile solution for training memory-augmented LMs.

会话问题应答（CQA）系统旨在为用户提供自然语言答案，以信息寻求对话。现有的CQA基准测试与预先收集的人类谈话进行比较模型，使用在会话历史中提供的地面真理答案。它仍然尚不清楚我们是否可以依赖于模型开发的这种静态评估，以及当前系统是否能够充分地概括为现实世界的人机对话。在这项工作中，我们开展了最先进的CQA系统的大规模人类评估，人类评估人员与模型交谈并判断了答案的正确性。我们发现，人机对话的分布与人类谈话的分配急剧不同，并且在模型排名方面存在人和金历史评估之间的分歧。我们进一步调查了如何改进自动评估，并提出基于预测历史的问题重写机制，与人类判断更好地相关。最后，我们讨论了各种建模策略和未来方向对更好的会话问题应答系统的影响。

This paper presents SimCSE, a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previous supervised counterparts. We find that dropout acts as minimal data augmentation, and removing it leads to a representation collapse. Then, we propose a supervised approach, which incorporates annotated pairs from natural language inference datasets into our contrastive learning framework by using "entailment" pairs as positives and "contradiction" pairs as hard negatives. We evaluate SimCSE on standard semantic textual similarity (STS) tasks, and our unsupervised and supervised models using BERT base achieve an average of 76.3% and 81.6% Spearman's correlation respectively, a 4.2% and 2.2% improvement compared to the previous best results. We also show-both theoretically and empirically-that the contrastive learning objective regularizes pre-trained embeddings' anisotropic space to be more uniform, and it better aligns positive pairs when supervised signals are available. 1 2 We randomly sample 10 6 sentences from English Wikipedia and fine-tune BERTbase with learning rate = 3e-5, N = 64. In all our experiments, no STS training sets are used.

petroni等。 （2019）证明，可以通过将它们表达为冻结式提示并将模型的预测准确性解释为下限，作为其编码的事实信息量的较低限制，从预先接收的语言模型中检索世界事实。随后的工作已经尝试通过搜索更好的提示来缩回估计，使用不相交的事实作为培训数据。在这项工作中，我们制作两个互补贡献，以更好地了解这些事实探测技术。首先，我们提出了OptiPrompt，一种新颖的和有效的方法，直接在连续嵌入空间中优化。我们发现这种简单的方法能够预测喇嘛基准中的额外6.4％的事实。其次，我们提出了一个更重要的问题：我们真的可以将这些探测结果解释为下限吗？这些提示搜索方法是否有可能从培训数据中学习？我们发现，有些令人惊讶的是，这些方法使用的培训数据包含了潜在的事实分布的某些规则，以及所有现有的提示方法，包括我们的方法，可以利用它们以获得更好的事实预测。我们开展一系列控制实验来解除“学习”从“学习召回”，提供了更详细的图片，不同的提示可以揭示关于预先接受的语言模型。

The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF-better few-shot fine-tuning of language models 1 -a suite of simple and complementary techniques for finetuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning. 2 * The first two authors contributed equally. 1 Alternatively, language models' best friends forever. 2 Our implementation is publicly available at https:// github.com/princeton-nlp/LM-BFF.